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Abstract 

Ubuntu ’s  Internet  Relay  Chat  teehnieal 
support  channel  has  hots  that  output 
specific  messages  in  response  to  com¬ 
mand  words  from  other  channel  users. 
These  messages  can  be  used  to  answer 
frequently-asked  questions  instead  of  re¬ 
quiring  an  expert  to  (repeatedly)  type  a 
lengthy  reply.  We  describe  an  approach  to 
automatically  distinguish  bot-answerable 
questions,  which  would  mitigate  this  prob¬ 
lem.  To  the  best  of  our  knowledge,  this  is 
the  first  work  on  investigating  question  an¬ 
swering  in  a  multiparticipant  chat  domain. 

Our  results  indicate  that  for  some  types  of 
questions,  supervised  learning  algorithms 
perform  well  on  this  task  and,  in  addition, 
that  character  n-grams  are  a  better  repre¬ 
sentation  than  traditional  bag-of-words  for 
this  task  and  domain. 

1  Introduction 

Ubuntu  (a  Linux-based  operating  system)  main¬ 
tains  multiple  Internet  Relay  Chat  (IRC)  chan¬ 
nels  for  technical  support.  Some  of  these  chan¬ 
nels  contain  bots,  which  are  automated  agents 
pre-programmed  to  perform  certain  tasks.  One 
of  the  bots  can  output  pre-written  messages, 
called  factoids,  in  response  to  command  words. 
For  example,  if  a  user  types  “!fiash”,  then 
the  bot  would  output  “To  install  Flash  see 
https://help.ubuntu... .mats/Flash  -  See  also  !Re- 
stricted  and  !Gnash”.  These  factoids  are  used  to 
answer  common  questions,  enforce  channel  guide¬ 
lines,  direct  non-English  speakers  (in  their  na¬ 
tive  tongue)  to  Ubuntu’s  foreign  language  support 
channels,  and  query  Ubuntu’s  repository  of  pack¬ 
ages.  While  useful,  this  bot  must  be  manually 
invoked.  Automating  the  bot  to  self-detect  and 
answer  questions  that  it  can  answer  could  reduce 
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the  workload  for  knowledgeable  experts  trying  to 
help  other  users.  This  is  applicable  to  not  only 
Ubuntu’s  technical  support  channels,  but  to  other 
IRC  channels  providing  technical  support  (e.g., 
Debian’s  support  channels)  and  to  channels  that 
use  similar  bots  (e.g.,  Eggdrop  and  Infobot)  for 
other  purposes. 

We  describe  initial  steps  on  a  self-invoking  bot. 
We  begin  by  investigating  the  multi-classification 
task  of  which  questions  are  bot-answerable  ques¬ 
tions  (BAQ)  and  which  are  human-answerable 
questions  (HAQ),  which  requires  a  human  to  an¬ 
swer  due  to  the  question’s  complexity.  We  imple¬ 
mented  a  baseline  non-learning  approach  and  su¬ 
pervised  support  vector  machines  (SVM)  and  k- 
nearest  neighbor  (fc-NN)  algorithms.  Our  results 
show  that  a  bot  can  answer  with  confidence  some 
types  of  questions,  especially  those  directing  users 
to  more  appropriate  channels  for  help  on  certain 
topics. 

Our  contributions  are  as  follows: 

•  Problem:  We  identified  a  real-world  problem 
that  has  not  been  investigated,  despite  bots 
having  been  around  for  years  on  IRC  chan¬ 
nels. 

•  Data:  We  created  an  annotated  multipartici¬ 
pant  chat  corpus  that  is  publicly  available. 

•  Empirical  study:  We  report  on  our  investi¬ 
gation  of  applying  supervised  learning  algo¬ 
rithms  and  leveraging  different  feature  rep¬ 
resentations,  whose  results  will  be  used  as  a 
foundation  for  a  larger  case-based  reasoning 
approach. 

•  Discussion:  We  describe  how  some  types 
of  automatically-answerable  questions  can  be 
easy  or  difficult  to  classify. 
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2  Related  Work 

Chat  is  a  difficult  medium  to  analyze:  its  charac¬ 
teristics  make  it  difficult  to  apply  traditional  nat¬ 
ural  language  processing  techniques.  It  has  un¬ 
common  features  such  as  frequent  use  of  abbrevia¬ 
tions,  acronyms,  missing  subject  pronouns,  emoti¬ 
cons,  abbreviated  nicknames,  words  stripped  of 
vowels  to  reduce  number  of  keystrokes,  and  entan¬ 
gled  conversation  threads  (Uthus  and  Aha,  2013a). 

In  the  multiparticipant  chat  domain,  there  has 
been  some  work  in  creating  a  hot  that  can  answer 
questions  with  Cobot  (Isbell  et  ah,  2006).  This  hot 
was  limited  in  capability  -  it  could  only  respond 
to  questions  directed  at  it.  Another  recent  work 
resulted  in  a  hot  which  could  respond  to  utter¬ 
ances  through  word  matching  and  used  templates 
for  output  (Shaikh  et  ah,  2010). 

Also  related  in  this  domain  are  a  few  military 
research  efforts  that  have  focused  on  classifying 
chat  messages.  One  examined  profile-driven  in- 
formalion  exfracfion  from  chaf  using  regular  ex¬ 
pressions  and  enfify  classes  (Berube  ef  ah,  2007). 
Anofher  examined  idenlifying  uncertainly  and  ur¬ 
gency  wilhin  a  chaf  message  using  rule-based  ap¬ 
proaches  and  sfafisfical  analysis  (Budlong  ef  ah, 
2009).  A  Ihird,  whose  work  is  mosl  similar  fo  ours, 
compared  several  supervised  algorifhms  for  classi¬ 
fying  chaf  ufferances  (Dela  Rosa  and  Ellen,  2009). 
Using  an  arlificial  chaf  log,  fhey  classified  mes¬ 
sages  as  eilher  non-imporlanf  filler  messages  or  as 
messages  confaining  Navy  ship  updafes.  Their  re- 
sulfs  showed  k-NN  and  SVM  performed  besf  for 
Ibis  fask.  Our  fask  differs  from  fhese  previous 
invesfigafions  in  fhaf  we  are  applying  supervised 
learning  algorifhms  fo  a  mulfi-labeled  corpus  com¬ 
posed  of  real  chaf  messages. 

This  problem  is  also  relafed  fo  fhe  larger  field 
of  quesfion  answering,  such  as  perfaining  fo  dis¬ 
cussion  boards  (Hong  and  Davison,  2009;  Kim  el 
ah,  2007),  frequenfly  asked  quesfion  files  (Burke 
el  ah,  1997),  and  communily -based  quesfion  an¬ 
swering  (Zhou  el  ah,  2012).  An  imporfanf  differ¬ 
ence  belween  Ibis  body  of  relafed  work  and  whal 
we  are  invesligafing  is  fhe  medium.  Mulliparlici- 
panl  chaf  is  more  difficull  fo  work  wilh  compared 
fo  ofher  mediums  due  fo  enlangled  conversalion 
Ihreads  -  a  researcher  cannol  easily  aulomafically 
analyze  fhe  messages  of  a  single  conversalion.  In 
prior  work,  researchers  could  usually  isolate  indi¬ 
vidual  conversations  aulomafically,  making  if  pos¬ 
sible  fo  idenlify  (fo  some  exlenl)  fhe  quesfion  and 
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Figure  1:  The  volume  of  messages  in  fhe  IRC 
channel  #ubuntu  during  2011. 


fhe  answer.  Anofher  imporfanf  difference  is  fhe 
temporal  scale  -  chaf  is  in  real-time,  and  a  chaf 
user  expecls  fo  receive  an  answer  quickly.  A  chaf 
user  can  only  see  messages  while  fhey  are  logged 
in  (in  fhe  case  where  Ihere  is  no  archive  being 
slored  offline).  Bolh  of  fhese  differences  resulfs 
in  new  challenges  nol  seen  in  ofher  mediums  for 
quesfion  answering. 

3  Ubuntu’s  IRC  Channels 

The  IRC  channel  tubuntu  is  Ubunlu’s  primary 
technical  supporl  channel.  If  provides  supporf 
for  Ihose  who  have  problems  using  Ubunlu;  if 
is  nol  used  for  socializing  or  for  receiving  help 
wilh  ofher  Linux  dislribufions  (e.g.,  Debian,  Linux 
Mini,  Fedora)  or  soflware. 

The  channel’s  Iraffic  level  varies  fhroughoul  fhe 
year  and  day  (see  Figure  1).  During  fhe  year,  if 
experiences  heavy  Iraffic  during  Ubunlu’s  semian¬ 
nual  new  releases  in  April  and  October  and  gen¬ 
erally  experiences  heavy  Iraffic  during  fhe  Norlh 
American  and  European  evening  hours.  Heavy 
Iraffic  creates  difficullies  for  users  frying  fo  gel  an¬ 
swers  to  Iheir  queslions. 

ubottu^  is  fhe  bof  lhal  can  access  1234  fac¬ 
toids,  corresponding  fo  2324  commands  (some 
factoids  are  mapped  fo  mulfiple  commands).  If 
can  also  provide  informafion  abouf  any  soflware 
package  found  in  Ubunlu’s  soflware  repository. 
A  channel  user  (oflenlimes  an  experl)  can  fask 
ubottu  to  answer  anofher  user’s  question  (see 

'http://ubottu.com/ 


[13:19]  <p5yx>  is  the  netbook 
remix  not  available  anymore? 
[13:20]  <histo>  !unr  |  p5yx 
[13:20]  <ubottu>  p5yx:  Starting 
with  Ubuntu  11.04,  the  Ubuntu 
Netbook  Edition  is  no  longer 
being  offered  as  a  separate 
install  as  Unity  is  now  standard 
for  all  Ubuntu  desktop  installs. 


Figure  3:  Distribution  of  the  sixty-eight  factoid  in¬ 
vocations  in  our  corpus. 


Figure  2:  Example  of  ubottu  being  invoked  with 
a  command  word  (in  this  case  “!unr”)  to  answer  a 
question. 

Figure  2).  Automating  ubottu  would  allow  ex¬ 
perts  to  focus  their  valuable  time  on  responding  to 
more  challenging  requests. 

4  Corpus 

We  created  an  annotated  corpus  by  pulling  ques¬ 
tions  from  the  Ubuntu  Chat  Corpus,  specifically 
from  the  #ubuntu  channel  logs  (Uthus  and  Aha, 
2013b).  This  corpus  has  4577  messages,  including 
2002  HAQs  and  2575  BAQs  from  68  factoid  cate¬ 
gories.  These  messages  were  taken  from  chat  logs 
from  28  April  201 1  (the  day  Ubuntu  1 1.04  was  re¬ 
leased)  to  13  October  2011  (the  day  before  Ubuntu 
11.10  was  released). 

We  looked  for  messages  in  which  a  question  is 
answered  with  a  factoid,  or  a  question  required 
a  human  to  answer.  To  judge  between  these  two 
types,  we  relied  on  the  expertise  of  users  and  how 
they  answered  the  questions.  To  reduce  noise,  we 
limited  HAQs  to  conversations  in  which  the  first 
reply  came  from  a  user  who  invoked  ubottu  fre¬ 
quently  (i.e.,  experts).  These  ubottu  invokers 
are  considered  a  better  judge  of  what  is  a  BAQ 
or  HAQ  compared  to  someone  who  rarely  invokes 
ubottu  to  answer  questions.  For  the  BAQs,  we 
restricted  questions  to  those  with  at  least  ten  ex¬ 
amples  mapped  to  a  factoid.  Figure  3  shows  the 
distribution  of  BAQs  to  factoids  in  our  corpus. 

Some  of  the  corpus’  messages  are  not  in  En¬ 
glish.  In  such  cases,  users  will  be  directed  to 
one  of  Ubuntu’s  foreign-language  support  chan¬ 
nels  (though  a  user  can  re-ask  their  question  in 
English).  Some  languages  present  in  the  corpus 
include  Spanish,  Erench,  Chinese,  Russian,  Ger¬ 
man,  Polish,  and  Portuguese.  Additionally,  some 
of  these  messages  are  written  with  non-Eatin  char¬ 
acters,  such  as  Chinese  and  Russian. 


5  Approach 

We  are  using  k-NN  and  SVM  algorithms  for 
classifying  messages.  This  builds  on  results  by 
Dela  Rosa  and  Ellen  (2009),  who  had  found  these 
two  supervised  learning  algorithms  to  work  best 
on  chat  messages.  Implementation  of  these  algo¬ 
rithms  were  obtained  from  Scikit-learn  (Pedregosa 
etal,  2011). 

Eor  preprocessing,  text  was  normalized  by  low¬ 
ering  the  case  for  each  term. 

We  examined  different  representations  for  en¬ 
coding  the  questions.  These  include  bag-of-words, 
bigrams,  and  character  n-grams.  With  the  charac¬ 
ter  n-grams,  we  examined  n-grams  which  overlap 
words  and  n-grams  which  are  restricted  to  within 
word  boundaries.  We  used  tf  —  idf  to  weigh  the 
features  and  for  feature  selection. 

6  Empirical  Study 

We  have  two  hypothesis  we  are  testing: 

HI:  Supervised  learning  algorithms  will  out¬ 
perform  a  non-learning  baseline  approach  for  clas¬ 
sifying  BAQs. 

H2:  Using  character  n-grams  for  this  domain 
will  allow  for  better  precision  and  recall  when 
compared  to  more  traditional  representations  of 
bag-of-words  and  bigrams. 

Our  intuition  for  H2  is  that  we  believe  that 
character  n-grams  will  allow  for  better  represen¬ 
tation  of  misspelled  words  commonly  seen  in  chat 
messages  when  compared  to  bag-of-words  and  bi¬ 
grams. 

6.1  Baseline 

Our  non-learning  baseline  algorithm  maps  ques¬ 
tions  to  factoids  by  checking  if  the  question  con¬ 
tains  the  factoid  command  as  a  word  token.  As  a 
reminder,  multiple  factoids  can  map  to  the  same 
response.  If  a  question  contains  multiple  factoids, 
then  the  most  frequently  invoked  factoid  is  applied 
(ties  are  broken  by  alphabetical  ordering).  If  a 


Representation 

Feature  Size 

Precision 

Recall 

Fo.5  Score 

Non-learning  baseline 

- 

- 

0.44 

0.24 

0.37 

SVM 

Character  3 -grams,  WB 

4000 

0.67 

0.42 

0.60 

Character  3 -grams 

3200 

0.67 

0.38 

0.59 

Character  4-grams,  WB 

3600 

0.63 

0.40 

0.57 

Bag-of- words 

1600 

0.62 

0.40 

0.56 

Character  4-grams 

4000 

0.58 

0.35 

0.51 

Bigrams 

1200 

0.51 

0.20 

0.39 

fc-NN 

Character  4-grams,  WB 

800 

0.55 

0.34 

0.49 

Character  3 -grams,  WB 

800 

0.55 

0.32 

0.48 

Character  4-grams 

1200 

0.57 

0.30 

0.48 

Character  3 -grams 

800 

0.54 

0.41 

0.47 

Bag-of- words 

400 

0.54 

0.31 

0.47 

Bigrams 

400 

0.44 

0.15 

0.32 

Table  1:  Results  for  the  baseline,  SVM  and  /c-NN  algorithms.  WB  means  the  eharaeter  n-grams  were 
bounded  within  word  boundaries. 


question  eontains  no  faetoids,  then  it  is  eonsidered 
aHAQ. 

6.2  Metrics 

We  used  a  10-fold  eross  evaluation  protoeol  and 
preeision,  reeall,  and  the  Fq.s  seore  as  our  evalu¬ 
ation  metries.  For  this  work,  we  eonsider  preei¬ 
sion  to  be  more  important  than  reeall  beeause  a 
hot  that  frequently  answers  questions  ineorreetly 
eould  anger  ehat  users  and  eause  them  to  ignore 
the  hots.  Therefore,  Fq.s  is  more  appropriate  here 
than  the  standard  F  score  because  it  places  more 
emphasis  on  precision.  Additionally,  as  these  are 
multi-classification  problems,  we  used  the  macro 
version  of  these  metrics  to  average  over  the  differ¬ 
ent  labels. 

When  calculating  precision,  recall,  and  Fq.s,  we 
omit  the  HAQ  scores  when  calculating  the  macro 
scores  for  this  multi-classification  problem.  We 
also  omitted  any  questions  that  are  incorrectly  la¬ 
beled  as  HAQ  for  calculating  precision  and  recall 
scores  because  a  HAQ  can  be  answered  by  a  hu¬ 
man  expert.  Essentially,  we  do  not  penalize  for 
erring  on  the  side  of  caution. 

6.3  Results 

Table  1  summarizes  the  results  of  the  application 
of  our  baseline  and  learning  algorithms.  We  ap¬ 


plied  all  variations  of  the  two  learning  algorithms, 
testing  on  all  combinations  of  representations  and 
feature  size  limits.  For  the  feature  size  lim¬ 
its,  we  tried  values  between  [400:4000]  in  steps  of 
400.  The  results  display  the  best  configuration  for 
each  representation. 

As  shown,  the  learning  algorithms  outper¬ 
formed  the  baseline  for  all  three  metrics,  support¬ 
ing  hypothesis  HI.  This  shows  that  some  ques¬ 
tions  cannot  be  easily  distinguished  by  simply 
looking  for  factoid  commands  within  the  ques¬ 
tions. 

In  regards  to  the  second  hypothesis,  both  learn¬ 
ing  algorithms  performed  best  when  using  char¬ 
acter  n-grams,  especially  when  restricted  by  word 
boundaries,  thus  supporting  H2.  We  believe  this 
is  due  to  the  character  n-grams  being  able  to  bet¬ 
ter  handle  noisy  nature  of  chat,  especially  with  the 
misspellings  and  abbreviations. 

We  next  examine  what  type  of  questions  do 
these  learning  algorithms  perform  well  on,  espe¬ 
cially  when  compared  to  our  baseline.  For  this, 
we  focus  on  the  results  of  applying  SVMs.  Figure 
4  compares  the  difference  of  Fq.s  scores  between 
SVM  and  the  baseline.  For  most  factoids,  SVMs 
performed  better  or  had  similar  performance  to 
the  baseline.  The  small  number  of  factoids  it 
performed  worse  on  were  generally  factoids  both 
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Figure  4:  Comparison  of  Fq.s  scores  between 
SVM  and  baseline. 

SVM  and  the  baseline  performed  poorly,  finding 
low  Fo.5  scores. 

Figure  5  shows  the  F0.5  scores  achieved  by 
SVM  for  each  individual  factoid  category.  These 
are  ordered  by  their  distribution  in  the  corpus  (see 
Figure  3).  One  fact  shown  by  this  figure  is  fhaf 
fhere  is  nof  a  sfrong  correlafion  befween  fhe  disfri- 
bufion  size  and  fhe  resulf  achieved  by  fhe  SVM. 
While  having  more  examples  wifhin  a  cafegory 
does  help,  fhere  are  plenfy  of  factoids  where  SVM 
performed  poorly.  This  shows  fhaf  if  is  fhe  dif- 
ficulfy  of  fhe  quesfions  fhemselves,  and  nof  fhe 
amounf  of  examples,  which  causes  difficulfy  for 
fhe  SVMs,  lef  alone  learning  algorifhms  for  fhis 
domain. 

One  sef  of  quesfions  SVMs  performs  well  on 
are  quesfions  where  users  are  subsequenfly  di- 
recfed  to  anofher  channel.  This  includes  Ubunfu’s 
non-English  channels  and  channels  fhaf  provide 
support  for  ofher  Linux  disfribufions.  SVMs  did 
well  on  all  fhe  non-English  facloids,  wifh  E0.5 
scores  ranging  befween  0.88  (for  Chinese)  and 
0.99  (for  Russian).  This  is  probably  due  to  fhese 
quesfions  having  uncommon  feafures,  such  as 
non-English  words  or  soflware  fhaf  is  nof  sup¬ 
posed  to  be  discussed  in  #ubuntu. 

One  similar  pair  of  quesfions,  which  are  ad¬ 
dressed  by  fwo  factoids,  caused  some  confusion 
for  fhe  learners  -  asking  for  permission  fo  ask  a 
quesfion  (e.g.,  “Can  I  ask  a  quesfion?”)  or  asking 
if  anyone  can  help  wifhouf  sfafing  fheir  problem 
(e.g.,  “Can  anyone  help  me?”).  This  commonly 
happens  wifh  firsf  fime  visitors  fo  fhe  channel,  as 
fhey  do  nof  know  fhe  channel  guidelines  and  will 
fhen  ask  for  permission  fo  ask  a  quesfion  or  if 
someone  could  help  fhem.  The  channel  operators 
fry  fo  encourage  users  fo  jusf  ask  fheir  quesfion 
-  this  happens  frequently  enough  that  there  are 
two  factoids  (labels  ask  and  anyone  in  the  corpus) 


Eigure  5:  Eq.s  scores  for  each  factoid  when  apply¬ 
ing  SVM,  ordered  by  distribution. 

to  answer  these  similar  questions.  Unfortunately, 
there  is  a  lack  of  consistency  in  invoking  these  two 
factoids,  and  as  such  the  learning  algorithms  we 
tested  had  difficulty  with  these  questions. 

Some  other  types  of  questions  that  SVMs  strug¬ 
gle  with  are  those  that  cover  a  wide-range  of  pos¬ 
sible  questions.  Eor  example,  #ubuntu  can  be  used 
either  in  cases  to  explain  what  the  channel  topic  is 
(for  those  asking),  or  to  get  users  on  topic  (with 
the  possible  off-topic  message  types  being  wide- 
ranging);  details,  which  can  be  used  whenever 
someone  asks  a  question  or  for  help  without  pro¬ 
viding  enough  details  for  anyone  to  begin  to  help; 
and  wine  to  help  users  with  problems  running  any 
type  of  Windows  program  under  Linux.  These 
types  of  questions  may  require  a  human  to  aid  in 
answering,  as  it  would  be  difficult  to  learn  all  pos¬ 
sible  types  of  questions  that  are  covered  by  these 
factoids. 

7  Conclusions 

We  have  investigated  applying  supervised  learning 
algorithms  to  classify  questions  as  HAQ  or  BAQ, 
and  our  results  show  that  these  algorithms  can  out¬ 
perform  a  non-learning  baseline  approach.  We 
also  show  that  character  n-grams  are  a  better  rep¬ 
resentation  than  traditional  bag-of-words  for  our 
task.  More  importantly,  the  learning  algorithms 
can  answer  some  types  of  questions  well,  indicat¬ 
ing  that  a  self-invoking  hot  can  be  created  that  can 
answer  common  questions  with  confidence. 

Eufure  work  fo  exfend  fhis  is  fo  apply  unsuper¬ 
vised  mefhods  for  finding  addifional  quesfions  fo 
mafch  wifh  fhe  facfoids.  This  would  greafly  exfend 
whaf  we  have  presenfed,  as  we  were  resfricfed  fo 
fhe  manually-labeled  messages  fo  mafch  quesfions 
wifh  answers.  We  plan  on  applying  a  case-based 
reasoning  framework  (Richfer  and  Weber,  2013) 
fo  achieve  such  a  goal. 


A  final  area  to  investigate  is  an  extension  of 
ubottu  that  ean  learn  to  update  its  knowledge. 
Currently,  only  a  few  users  are  allowed  to  edit  or 
add  new  faetoids  to  ubottu.  It  would  be  advanta¬ 
geous  if  it  eould  add  new  commands  and  factoids 
itself  by  summarizing  common  answers,  or  update 
outdated  factoids  should  it  see  a  common  pattern 
of  answers  conflicting  with  its  knowledge. 

Acknowledgments 

Thanks  to  NRL  for  funding  this  research.  David 
Uthus  performed  this  work  while  an  NRC  post¬ 
doctoral  fellow  located  at  the  Naval  Research  Lab¬ 
oratory.  The  views  and  opinions  contained  in  this 
paper  are  those  of  the  authors  and  should  not  be  in¬ 
terpreted  as  representing  the  official  views  or  poli¬ 
cies,  either  expressed  or  implied,  of  NRL  or  the 
DoD. 

References 

Christopher  D.  Berube,  Janet  M.  Hitzeman,  Roderick  J. 
Holland,  Robert  L.  Anapol,  and  Stephen  R.  Moore. 
2007.  Supporting  chat  exploitation  in  DoD  enter¬ 
prises.  In  Proceedings  of  the  International  Com¬ 
mand  and  Control  Research  and  Technology  Sym¬ 
posium.  CCRR 

Emily  R.  Budlong,  Sharon  M.  Walter,  and  Ozgur  Yil- 
mazel.  2009.  Recognizing  connotative  meaning  in 
military  chat  communications.  In  Proceedings  of 
Evolutionary  and  Bio-Inspired  Computation:  The¬ 
ory  and  Applications  III.  SPIE. 

Robin  D.  Burke,  Kristian  J.  Hammond,  Vladimir  Ku- 
lyukin,  Steven  L.  Lytinen,  Noriko  Tomuro,  and  Scott 
Schoenberg.  1997.  Question  answering  from  fre¬ 
quently  asked  question  files:  Experiences  with  the 
FAQ  FINDER  system.  AI Magazine,  18(2). 

Kevin  Dela  Rosa  and  Jeffrey  Ellen.  2009.  Text  clas¬ 
sification  methodologies  applied  to  micro-text  in 
military  chat.  In  Proceedings  of  the  International 
Conference  on  Machine  Learning  and  Applications, 
pages  710-714.  IEEE  Computer  Society. 

Liangjie  Hong  and  Brian  D.  Davison.  2009.  A 
classification-based  approach  to  question  answering 
in  discussion  boards.  In  Proceedings  of  the  32nd 
International  ACM  SIGIR  Conference  on  Research 
and  Development  in  Information  Retrieval,  pages 
171-178.  ACM. 

Charles  Lee  Isbell,  Michael  Kearns,  Satinder  Singh, 
Christian  R.  Shelton,  Peter  Stone,  and  Dave  Kor- 
mann.  2006.  Cobot  in  LambdaMOO:  An  adap¬ 
tive  social  statistics  agent.  Autonomous  Agents  and 
Multi-Agent  Systems,  13(3):327-354. 


Jihie  Kim,  Erin  Shaw,  Grace  Chern,  and  Donghui  Eeng. 
2007.  An  intelligent  discussion-hot  for  guiding  stu¬ 
dent  interactions  in  threaded  discussions.  In  Pro¬ 
ceedings  of  the  AAAI  Spring  Symposium  on  Interac¬ 
tion  Challenges  for  Intelligent  Assistants .  AAAI. 

Eabian  Pedregosa,  Gael  Varoquaux,  Alexandre  Gram- 
fort,  Vincent  Michel,  Bertrand  Thirion,  Olivier 
Grisel,  Mathieu  Blondel,  Peter  Prettenhofer,  Ron 
Weiss,  Vincent  Dubourg,  Jake  Vanderplas,  Alexan¬ 
dre  Passos,  David  Cournapeau,  Matthieu  Bmcher, 
Matthieu  Perrot,  and  Edouard  Duchesnay.  2011. 
Scikit-learn:  Machine  Learning  in  Python.  Journal 
of  Machine  Learning  Research,  12:2825-2830. 

Michael  M.  Richter  and  Rosina  O.  Weber.  2013.  Case- 
Based  Reasoning:  A  Textbook.  Springer  Berlin. 

Samira  Shaikh,  Tomek  Strzalkowski,  Aaron  Broad- 
well,  Jennifer  Stromer-Galley,  Sarah  Taylor,  and 
Nick  Webb.  2010.  MPC:  A  multi-party  chat  cor¬ 
pus  for  modeling  social  phenomena  in  discourse.  In 
Proceedings  of  the  Seventh  conference  on  Interna¬ 
tional  Language  Resources  and  Evaluation,  pages 
2007-2013.  European  Language  Resources  Associ¬ 
ation. 

David  C.  Uthus  and  David  W.  Aha.  2013a.  Multipar¬ 
ticipant  chat  analysis:  A  survey.  Artificial  Intelli¬ 
gence,  199-200:106-121. 

David  C.  Uthus  and  David  W.  Aha.  2013b.  The 
Ubuntu  Chat  Corpus  for  multiparticipant  chat  anal¬ 
ysis.  In  Proceedings  of  the  AAAI  Spring  Symposium 
on  Analyzing  Microtext,  pages  99-102.  AAAI. 

Tom  Chao  Zhou,  Michael  R.  Lyu,  and  Irwin  King. 
2012.  A  classification-based  approach  to  question 
routing  in  community  question  answering.  In  Pro¬ 
ceedings  of  the  21st  International  Conference  Com¬ 
panion  on  World  Wide  Web,  pages  783-790.  ACM. 


